Disrupted orbital order and the pseudo-gap in layered 1T-TaS2_2

We present a state-of-the-art density functional theory (DFT) study which models crucial features of the partially disordered orbital order stacking in the prototypical layered transition metal dichalcogenide 1T-TaS2 . Our results not only show that DFT models with realistic assumptions about the orbital order perpendicular to the layers yield band structures which agree remarkably well with experiments. They also demonstrate that DFT correctly predicts the formation of an excitation pseudo-gap which is commonly attributed to Mott-Hubbard type electron-electron correlations. These results highlight the importance of interlayer interactions in layered transition metal dichalcogenides and serve as an intriguing example of how disorder within an electronic crystal can give rise to pseudo-gap features

Total Denoising: Unsupervised Learning of 3D Point Cloud Cleaning

We show that denoising of 3D point clouds can be learned unsupervised, directly from noisy 3D point cloud data only. This is achieved by extending recent ideas from learning of unsupervised image denoisers to unstructured 3D point clouds. Unsupervised image denoisers operate under the assumption that a noisy pixel observation is a random realization of a distribution around a clean pixel value, which allows appropriate learning on this distribution to eventually converge to the correct value. Regrettably, this assumption is not valid for unstructured points: 3D point clouds are subject to total noise, i. e., deviations in all coordinates, with no reliable pixel grid. Thus, an observation can be the realization of an entire manifold of clean 3D points, which makes a na\"ive extension of unsupervised image denoisers to 3D point clouds impractical. Overcoming this, we introduce a spatial prior term, that steers converges to the unique closest out of the many possible modes on a manifold. Our results demonstrate unsupervised denoising performance similar to that of supervised learning with clean data when given enough training examples - whereby we do not need any pairs of noisy and clean training data.Comment: Proceedings of ICCV 201

Single-image Tomography: 3D Volumes from 2D Cranial X-Rays

As many different 3D volumes could produce the same 2D x-ray image, inverting this process is challenging. We show that recent deep learning-based convolutional neural networks can solve this task. As the main challenge in learning is the sheer amount of data created when extending the 2D image into a 3D volume, we suggest firstly to learn a coarse, fixed-resolution volume which is then fused in a second step with the input x-ray into a high-resolution volume. To train and validate our approach we introduce a new dataset that comprises of close to half a million computer-simulated 2D x-ray images of 3D volumes scanned from 175 mammalian species. Applications of our approach include stereoscopic rendering of legacy x-ray images, re-rendering of x-rays including changes of illumination, view pose or geometry. Our evaluation includes comparison to previous tomography work, previous learning methods using our data, a user study and application to a set of real x-rays

Joint Material and Illumination Estimation from Photo Sets in the Wild

Faithful manipulation of shape, material, and illumination in 2D Internet images would greatly benefit from a reliable factorization of appearance into material (i.e., diffuse and specular) and illumination (i.e., environment maps). On the one hand, current methods that produce very high fidelity results, typically require controlled settings, expensive devices, or significant manual effort. To the other hand, methods that are automatic and work on 'in the wild' Internet images, often extract only low-frequency lighting or diffuse materials. In this work, we propose to make use of a set of photographs in order to jointly estimate the non-diffuse materials and sharp lighting in an uncontrolled setting. Our key observation is that seeing multiple instances of the same material under different illumination (i.e., environment), and different materials under the same illumination provide valuable constraints that can be exploited to yield a high-quality solution (i.e., specular materials and environment illumination) for all the observed materials and environments. Similar constraints also arise when observing multiple materials in a single environment, or a single material across multiple environments. The core of this approach is an optimization procedure that uses two neural networks that are trained on synthetic images to predict good gradients in parametric space given observation of reflected light. We evaluate our method on a range of synthetic and real examples to generate high-quality estimates, qualitatively compare our results against state-of-the-art alternatives via a user study, and demonstrate photo-consistent image manipulation that is otherwise very challenging to achieve

Metappearance: Meta-Learning for Visual Appearance Reproduction

There currently are two main approaches to reproducing visual appearance using Machine Learning (ML): The first is training models that generalize over different instances of a problem, e.g., different images from a dataset. Such models learn priors over the data corpus and use this knowledge to provide fast inference with little input, often as a one-shot operation. However, this generality comes at the cost of fidelity, as such methods often struggle to achieve the final quality required. The second approach does not train a model that generalizes across the data, but overfits to a single instance of a problem, e.g., a flash image of a material. This produces detailed and high-quality results, but requires time-consuming training and is, as mere non-linear function fitting, unable to exploit previous experience. Techniques such as fine-tuning or auto-decoders combine both approaches but are sequential and rely on per-exemplar optimization. We suggest to combine both techniques end-to-end using meta-learning: We over-fit onto a single problem instance in an inner loop, while also learning how to do so efficiently in an outer-loop that builds intuition over many optimization runs. We demonstrate this concept to be versatile and efficient, applying it to RGB textures, Bi-directional Reflectance Distribution Functions (BRDFs), or Spatially-varying BRDFs (svBRDFs)

Plateau-reduced Differentiable Path Tracing

Current differentiable renderers provide light transport gradients with respect to arbitrary scene parameters. However, the mere existence of these gradients does not guarantee useful update steps in an optimization. Instead, inverse rendering might not converge due to inherent plateaus, i.e., regions of zero gradient, in the objective function. We propose to alleviate this by convolving the high-dimensional rendering function that maps scene parameters to images with an additional kernel that blurs the parameter space. We describe two Monte Carlo estimators to compute plateau-free gradients efficiently, i.e., with low variance, and show that these translate into net-gains in optimization error and runtime performance. Our approach is a straightforward extension to both black-box and differentiable renderers and enables optimization of problems with intricate light transport, such as caustics or global illumination, that existing differentiable renderers do not converge on.Comment: Accepted to CVPR 2023. Project page and interactive demos at https://mfischer-ucl.github.io/prdpt

3D Object Class Detection in the Wild

Object class detection has been a synonym for 2D bounding box localization for the longest time, fueled by the success of powerful statistical learning techniques, combined with robust image representations. Only recently, there has been a growing interest in revisiting the promise of computer vision from the early days: to precisely delineate the contents of a visual scene, object by object, in 3D. In this paper, we draw from recent advances in object detection and 2D-3D object lifting in order to design an object class detector that is particularly tailored towards 3D object class detection. Our 3D object class detection method consists of several stages gradually enriching the object detection output with object viewpoint, keypoints and 3D shape estimates. Following careful design, in each stage it constantly improves the performance and achieves state-ofthe-art performance in simultaneous 2D bounding box and viewpoint estimation on the challenging Pascal3D+ dataset

Deep Reflectance Maps

Undoing the image formation process and therefore decomposing appearance into its intrinsic properties is a challenging task due to the under-constraint nature of this inverse problem. While significant progress has been made on inferring shape, materials and illumination from images only, progress in an unconstrained setting is still limited. We propose a convolutional neural architecture to estimate reflectance maps of specular materials in natural lighting conditions. We achieve this in an end-to-end learning formulation that directly predicts a reflectance map from the image itself. We show how to improve estimates by facilitating additional supervision in an indirect scheme that first predicts surface orientation and afterwards predicts the reflectance map by a learning-based sparse data interpolation. In order to analyze performance on this difficult task, we propose a new challenge of Specular MAterials on SHapes with complex IllumiNation (SMASHINg) using both synthetic and real images. Furthermore, we show the application of our method to a range of image-based editing tasks on real images.Comment: project page: http://homes.esat.kuleuven.be/~krematas/DRM

Electronic self-organization in layered transition metal dichalcogenides

The interplay between different self-organized electronically ordered states and their relation to unconventional electronic properties like superconductivity constitutes one of the most exciting challenges of modern condensed matter physics. In the present thesis this issue is thoroughly investigated for the prototypical layered material 1T-TaS2 both experimentally and theoretically. At first the static charge density wave order in 1T-TaS2 is investigated as a function of pressure and temperature by means of X-ray diffraction. These data indeed reveal that the superconductivity in this material coexists with an inhomogeneous charge density wave on a macroscopic scale in real space. This result is fundamentally different from a previously proposed separation of superconducting and insulating regions in real space. Furthermore, the X-ray diffraction data uncover the important role of interlayer correlations in 1T-TaS2. Based on the detailed insights into the charge density wave structure obtained by the X-ray diffraction experiments, density functional theory models are deduced in order to describe the electronic structure of 1T-TaS2 in the second part of this thesis. As opposed to most previous studies, these calculations take the three-dimensional character of the charge density wave into account. Indeed the electronic structure calculations uncover complex orbital textures, which are interwoven with the charge density wave order and cause dramatic differences in the electronic structure depending on the alignment of the orbitals between neighboring layers. Furthermore, it is demonstrated that these orbital-mediated effects provide a route to drive semiconductor-to-metal transitions with technologically pertinent gaps and on ultrafast timescales. These results are particularly relevant for the ongoing development of novel, miniaturized and ultrafast devices based on layered transition metal dichalcogenides. The discovery of orbital textures also helps to explain a number of long-standing puzzles concerning the electronic self-organization in 1T-TaS2 : the ultrafast response to optical excitations, the high sensitivity to pressure as well as a mysterious commensurate phase that is commonly thought to be a special phase a so-called “Mott phase” and that is not found in any other isostructural modification